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Abstract 

The  Department  of  Veterans  Affairs  (DVA)  is  faced  with  the 
major  challenge  of  converting  the  nation's  largest  integrated 
health  system  from  one  focused  on  "ill  care"  to  one  focused  on 
"well  care".  When  changes  are  made  in  structure  and  process 
elements  of  healthcare  delivery,  a  system  of  measurement  must  be 
developed  to  demonstrate  that  outcomes  have  not  been  negatively 
impacted  by  these  changes. 

Although  the  concept  of  each  patient  being  assigned  to  one 
Primary  Care  Provider  (PCP)  has  been  in  place  at  the  Cheyenne 
Veterans  Affairs  Medical  Center  (VAMC)  for  some  time,  inclusion 
of  ancillary  staff  to  complete  a  comprehensive  primary  care  team 
has  only  recently  been  implemented  (November  30,  1998)  .  The 
purpose  of  this  project  was  twofold:  to  assess  whether 
sufficient  reliable  and  valid  data  exist  upon  which  to  base  a 
comparison  of  outcomes  pre-  and  post-primary  care  team 
implementation;  and  if  so,  to  make  such  a  comparison. 

A  para-analysis  following  the  Joint  Commission  on 
Accreditation  of  Healthcare  Organizations  (JCAHO)  Clinical  Value 
Compass  approach  was  chosen  to  assess  whether  the  implementation 
of  primary  care  teams  added  value  to  the  healthcare  provided  by 
the  Cheyenne  VAMC. 

Potential  outcomes  measurements  were  identified  via  review 


of  the  literature.  A  panel  of  persons  involved  in  the  local 
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collection  and  interpretation  of  data  reviewed  the  list  of 
identified  measurements,  assisted  in  determining  which  measures 
were  readily  available  and  interpretable,  and  ensured  that  only 
valid  and  practical  measures  were  selected. 

Data  were  extracted  from  customer  satisfaction  survey 
results,  directors'  performance  measures,  fiscal  reports,  and 
various  ad  hoc  reports  to  compare  outcomes  during  the  pre-  and 
post-implementation  timeframes. 

Reliable,  valid  outcomes  measures  were  identified  among 
currently  collected  information.  Outcomes  in  all  four  of  the 
cardinal  points  of  the  Clinical  Value  Compass  were  improved  or 
unchanged.  However,  whether  these  changes  in  outcomes  can  be 
attributed  to  primary  care  team  implementation  may  be 
questioned.  Since  the  healthcare  environment  is  far  from  static, 
other  changes  may  have  contributed  to  differences  seen  in 
selected  outcomes  measures. 

Review  of  the  valid,  reliable  measures  identified  in  this 
project  could  be  cont inued/expanded  to  measure  success  or  lack 
thereof  for  other  new  programs,  in  addition  to  ongoing  review  of 
primary  care.  Results  could  be  applied  to  VAMCs  of  similar  size 
and  scope,  and  the  study  could  be  replicated  at  other  size  and 
scope  institutions  for  comparison/applicability.  Replication  of 
this  study  at  an  institution  of  similar  scope,  but  lacking 
primary  care  teams,  would  lead  to  more  definite  conclusions. 
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Introduction 

In  recent  years  our  nation' s  healthcare  system  has 
undergone  a  major  transition,  shifting  from  a  focus  of  "ill 
care"  to  one  of  "well  care".  Spending  for  healthcare  has  risen 
at  a  rate  far  exceeding  that  attributable  to  inflation.  Economic 
constraints  on  healthcare  spending  have  ensued.  System-wide 
changes  in  healthcare  delivery  are  a  result  of  both  the  shift  in 
focus  of  care  and  the  economic/political  climate. 

Effectiveness  in  healthcare  provision  is  dependent  upon 
wise  stewardship  of  scarce  healthcare  resources.  The 
administrator  faces  the  challenge  of  acting  business-like  in 
order  to  maintain  fiscal  solvency,  while  never  losing  sight  of 
the  patient  as  the  central  mission  focus.  Clinicians  and 
administrators  may  find  themselves  at  loggerheads  when  trying  to 
balance  the  iron  triangle  of  cost,  quality,  and  access. 

The  Department  of  Veterans  Affairs  (DVA)  is  faced  with  the 
major  challenge  of  converting  the  nation's  largest  integrated 
health  system  from  one  focused  on  "ill  care"  to  one  focused  on 
"well  care".  As  healthcare's  focus  changes  to  one  of  wellness, 
the  DVA  is  finding  that  current  distribution  of  staffing  and 
facilities  frequently  does  not  match  the  needs  of  the 
demographically  changed  population.  Turf  protectionism  and 
political  special  interests  make  change  difficult  in  the  federal 
system.  Against  these  odds,  the  DVA  has  undergone  unprecedented 
change,  reinventing  itself  to  meet  the  changing  needs  of  the 
population  it  serves. 
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When  changes  are  made  in  structure  and  process  elements  of 
healthcare  delivery,  a  system  of  measurement  must  be  developed 
to  demonstrate  that  outcomes  have  not  been  negatively  impacted 
by  these  changes. 

Conditions  which  Prompted  the  Study 

Although  the  concept  of  each  patient  being  assigned  to  one 
primary  care  provider  (PCP)  has  been  in  place  at  the  Cheyenne 
Veterans  Affairs  Medical  Center  (VAMC)  for  some  time,  inclusion 
of  ancillary  staff  to  complete  a  comprehensive  primary  care  team 
has  only  recently  been  implemented  (November  30,  1998)  .  The 
primary  care  teams  are  multidisciplinary,  including  designated 
pharmacists,  social  workers,  licensed  practical  nurses,  and 
administrative  support  personnel,  in  addition  to  registered 
nurse  case  managers  and  primary  care  providers . 

Data  that  may  allow  comparison  of  structure,  process,  and 
outcomes  before  and  after  the  implementation  of  primary  care 
teams  are  available  through  Decision  Support  Service  (DSS) , 
Director's  Performance  Measures,  cost  and  staffing  analyses, 
various  ad  hoc  reports,  and  customer  satisfaction  surveys. 

Statement  of  the  Problem  or  Question 

Extensive  changes  have  been  made  in  ambulatory  care 
structure  and  process.  It  is  important  to  define  those  changes, 
and  to  attempt  to  measure  their  impact  on  cost,  access,  and 
quality . 
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Literature  Review 


Structure,  Process,  Outcomes 

In  1980,  Avedis  Donabedian  identified  and  defined 
structure,  process,  and  outcome  as  the  three  categories  of 
variables  within  quality  assessment  (Hill,  1997).  More  recently, 
this  time-honored  structure-process-outcome  framework  has  been 
incorporated  into  a  dynamic  model,  recognizing  feedback  among 
clients,  the  system  in  which  care  is  provided,  and  interventions 
(Mitchell  et  al,  1998). 

Paul  Ellwood,  M.D.,  in  his  famous  1988  Shattuck  Lecture, 
acknowledged  that  managed  care  has  democratized  the  healthcare 
system.  At  the  same  time,  he  expressed  his  concern  about  the 
lack  of  a  consistent  methodology  designed  to  measure  effects  of 
the  choices  made  by  patients,  payors,  and  physicians.  Dr. 

Ellwood  proposed  systematic  outcomes  management  as  a  means  to 
bring  quality  and  order  to  the  nation' s  chaotic  healthcare 
system  (Ellwood,  1988). 

Early  quality  improvement  (QI)  efforts  were  focused 
predominantly  on  improving  structure  and  process,  assuming  that 
improved  outcomes  would  follow.  As  the  healthcare  world  has 
evolved,  QI  has  adapted.  In  the  current  competitive  climate, 
outcomes  are  being  used  to  compare  multiple  aspects  of 
healthcare.  Although  it  is  clear  that  outcomes  measurement  is 
important,  controversy  exists  with  regard  to  selection  of 
outcomes  measures.  The  challenge  is  to  select  outcomes  that  are 
comprehensive,  comparable,  and  meaningful  (Kleinpell,  1997). 
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Concerns 

There  is  demand  for  improved  outcomes  across  the 
healthcare  continuum.  Clinicians  and  administrators  want  to  have 
data  quickly,  but  some  program  objectives  cannot  be  measured  in 
the  short  term.  Events  such  as  mortality  may  be  too  rare  to 
track.  Risk  factors  have  been  used  as  surrogate  measures  of 
mortality,  and  preventative  health  measures  often  serve  as 
surrogate  measures  of  population  health.  The  pressure  to 
demonstrate  accountability  through  outcomes  has  led  to 
development  of  a  broad  range  of  measurement  initiatives  (DeWolf 
&  Giloth,  1998) . 

Confusion  exists  regarding  the  distinction  between  health 
outcomes  and  performance  measures.  Health  outcomes  measures  are 
indicators  of  population  health  and  have  been  defined  as  crude 
rates  of  adverse  events  within  a  population.  Performance 
indicators,  on  the  other  hand,  are  specific  to  those  aspects  of 
care  that  can  be  altered  by  the  staff  whose  performance  is  being 
monitored  (Giuffrida,  Gravelle,  &  Roland,  1999). 

Performance  measures  have  become  key  assessment  factors 
reviewed  by  third  party  payors  in  the  managed  care  environment. 
Interest  in  measuring  outcomes  has  grown  out  of  fears  that 
performance  measures  emphasize  cost  containment  without 
measuring  quality.  The  routine  assessment  of  outcomes  is 
essential  to  demonstrate  and  ensure  the  quality  of  care  provided 
(Reemtsma  &  Morgan,  1997) . 

The  outcomes  momentum  of  late  has  led  to  neglect  of  the 
use  of  process  measures  to  assess  the  quality  of  care.  Process 
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measures  are  amenable  to  direct  measurement,  are  readily 
interpreted,  and  can  identify  deficiencies  in  care  processes 
that  need  to  be  remedied.  The  disadvantage  to  using  process 
measures  is  that  there  is  often  a  lack  of  evidence  linking 
processes  of  care  to  desired  outcomes.  Quality  assessment  must 
take  a  balanced  approach,  relying  on  measurement  of  both 
processes  and  outcomes  (Crombie  &  Davies,  1998). 

Outcomes  measurement  routinely  relies  on  retrospective 
analysis  of  administrative  databases.  The  usefulness  of  any  data 
is  limited  by  its  quality  and  completeness.  Adequate  assessment 
of  many  aspects  of  care  may  not  be  possible  with  currently 
available  data  (Ray,  1997) .  It  has  been  recommended  that 
disease-specific,  patient  care-centered  outcomes  measures  be 
included  in  administrative  databases.  However,  the  logistics  of 
implementing  this  recommendation  are  formidable,  and  the 
benefits  to  patient  care  are  questionable  (Liang,  &  Shadick, 

1997)  . 

Data,  in  order  to  be  useful,  must  be  transformed  into 
information.  A  decision  support  system  can  be  a  useful  tool  in 
this  endeavor.  Such  a  system  goes  beyond  performance  measurement 
in  that  it  can  help  to  identify  causes  of  care  process  problems, 
potentiating  improved  outcomes  (Brailer  et  al,  1996) . 

A  study  was  undertaken  in  1996  to  validate  the  DVA' s 
decision  support  system  data.  Study  results  showed  a  high 
correlation  between  the  numbers  of  hospital  stays  reported  in 
the  patient  treatment  file  and  the  data.  Aggregate  cost  data 
were  compared  with  Medicare  relative  resource  weights  for  all 
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discharged  patients  in  given  diagnosis  related  groups  (DRGs) .  A 
high  correlation  coefficient  (0.853)  was  calculated.  The 
majority  of  ambulatory  care  encounters  reported  in  the 
ambulatory  care  database  were  also  reported  in  the  decision 
support  system  at  all  sites  studied,  but  statistical  power  of 
the  correlation  varied.  The  authors  concluded  that  the  decision 
support  system  has  great  promise,  but  that  its  validity  hinges 
on  the  accuracy  of  utilization  data,  which  may  currently  be 
incompletely  recorded.  Problems  were  also  noted  with  access  to 
data  and  with  the  ability  to  distinguish  long-term  from  acute 
hospital  care  (Barnett  &  Rodgers,  1999)  . 

Primary  Care/Case  Management 

The  Institute  of  Medicine  defines  primary  care  as  "the 
provision  of  integrated,  accessible  health  care  services  by 
clinicians  who  are  accountable  for  addressing  a  large  majority 
of  personal  health  care  needs,  developing  a  sustained 
partnership  with  patients,  and  practicing  in  the  context  of 
family  and  community"  (National  Academy  of  Sciences,  1999) . 

A  nurse  case  manager  is  ordinarily  a  key  member  of  an 
interdisciplinary  primary  care  team.  The  nurse-coordinated  case 
management  model  of  care  delivery  is  intended  to  promote 
accountability  and  cost-effectiveness.  A  study  of  nurse  case- 
managed  patients  showed  a  72%  reduction  in  acute  and  skilled 
inpatient  bed  days  of  care  and  significant  cost  savings  (Ford, 


1999)  . 
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The  pharmacist's  role  in  primary  care  often  includes 
monitoring  of  therapeutic  outcomes  in  addition  to  provision  of 
medication-related  counseling  to  patients.  Pharmacists  can 
provide  medical  triage  through  appropriate  patient  referral  upon 
detection  of  drug  and  disease-related  adverse  events.  In  a  study 
of  a  program  designed  to  ensure  a  consistent  approach  to 
diabetes  management  and  education,  health  outcomes  of  older 
patients  with  diabetes  were  improved  by  pharmacist-conducted 
counseling.  Involvement  as  counselors  enabled  pharmacists  to 
detect  progressing  adverse  events  and  intervene,  averting  more 
severe  sequelae  (Baran  et  al .  ,  1999)  . 

A  1995  study  compared  outcomes  and  costs  of  care  for  acute 
low  back  pain  among  patients  seen  by  primary  care  practitioners, 
chiropractors,  and  orthopedic  surgeons.  Clinical  outcomes  did 
not  differ  significantly  among  the  three  groups.  However, 
patient  satisfaction  was  significantly  higher  among  those  cared 
for  by  chiropractors,  and  costs  of  care  were  lowest  for  patients 
of  primary  care  practitioners  (Carey  et  al,  1995). 

Studies  have  indicated  that  income  disparity  is  positively 
correlated  with  health  disparity.  However,  access  to  primary 
care  services  reduces  the  adverse  impact  of  poverty  on  health 
outcomes  (Starfield,  1999) .  This  is  particularly  germane  to  the 
DVA,  as  those  who  use  the  DVA  healthcare  system  report  lower 
average  annual  incomes  and  poorer  health  status  than  the  general 
population  (Topping  &  Ginter,  1998). 

In  a  two-year  clinical  trial  at  the  VAMC  in  Memphis,  TN, 
128  veterans  were  randomized  to  receive  either  interdisciplinary 
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primary  care  Geriatric  Evaluation  and  Management  or  usual  care. 
Outcome  measures  included  health  status,  function,  quality  of 
life,  and  mortality.  Researchers  concluded  that  a  comprehensive 
interdisciplinary  approach  to  primary  care  may  significantly 
improve  outcomes  for  targeted  older  adults,  and  that  these 
outcomes  may  continue  to  improve  over  time  (Burns,  Nicolas, 

Mart indale-Adams ,  &  Graney,  2000). 

However,  quite  different  conclusions  were  reached  by 
researchers  at  the  VAMC,  Durham,  NC,  who  conducted  a  multi-site 
randomized,  controlled  trial  of  enhanced  access  to  primary  care 
for  patients  with  congestive  heart  failure.  In  this  trial, 
enhanced  access  to  primary  care  did  not  improve  patients'  self- 
reported  health  status,  and  was  associated  with  more  frequent 
hospitalizations  (Oddone,  Weinberger,  Giobbie-Hurder ,  Landsman, 

&  Henderson,  1999). 

Selection  of  Outcomes  Measures 

Outcomes  categories  recommended  for  review  include  clinical 
and  functional  outcomes,  charges,  cost  and  effectiveness  data, 
and  complications  of  treatment,  as  well  as  health  related 
quality  of  life  and  patient  satisfaction  (Reemtsma  &  Morgan, 

1997) .  A  more  broad  classification  delineates  three  categories 
of  outcomes  analysis:  clinical,  humanistic,  and  economic 
(Generali,  1999) . 

A  time-related  concept  of  outcomes  measurement  has  been 
suggested  as  potentially  more  applicable  and  acceptable  to 
providers  and  patients  in  a  primary  care  setting.  The  three 
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proposed  time  classifications  and  some  recommended  outcomes 
measures  associated  with  each  include:  short-term/patient 
satisfaction;  intermediate/compliance;  and  long-term/symptom 
resolution,  health  status,  and  quality  of  life  (Stott, 
Kinnersley  &  Elwyn,  1997)  . 

Health-adjusted  life  expectancy  has  been  proposed  as  an 
aggregate  outcome  measure.  A  long-term  strategy  to  increase 
health-adjusted  life  expectancy  would  include  development  of  an 
operational  definition  of  population  health  and  alignment  of 
financial  incentives  with  the  goal  of  population  health 
improvement  (Kindig,  1998) . 

The  Joint  Commission  on  Accreditation  of  Healthcare 
Organizations  (JCAHO)  has  copyrighted  a  Clinical  Value  Compass 
Worksheet.  The  compass  has  four  cardinal  points,  corresponding 
to  four  categories  of  outcomes:  (1)  functional  status,  risk 
status,  and  well  being;  (2)  costs;  (3)  satisfaction  with 
healthcare  and  perceived  benefit;  and  (4)  clinical  outcomes. 
Employing  the  Clinical  Value  Compass  approach  for  improving  the 
value  of  healthcare  services  includes  measuring  the  value  of 
care  provided  to  similar  patient  populations,  analyzing  care 
delivery  processes,  testing  of  changed  delivery  processes,  and 
determining  whether  changes  led  to  better  outcomes  and/or  lower 
costs  (Nelson  et  al,  1996) .  Figure  one  displays  the  JCAHO 
Clinical  Value  Compass. 
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Figure  1 .  JCAHO  Clinical  Value  Compass 

Some  of  the  variables  routinely  cited  in  the  literature  as 
potential  indicators  of  healthcare  outcomes  include: 

•  Percentage  of  patients  seen  by  designated  primary  care 
provider 

•  Length  of  stay 

•  Total  discharges  per  patient  population 

•  Unscheduled  admissions 

•  ICU  admissions 

•  Re-admission  within  x  days  of  discharge 

•  Total  ambulatory  visits  per  patient 

•  Unscheduled  ambulatory  or  emergency  room  visits 

•  Patient  satisfaction/perception  of  health  status 

•  Length  of  office  visit 
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•  Staff  satisfaction 

•  Number  of  patients  seen  per  unit  of  time  or  per  number  of  full 
time  employee  equivalents  assigned 

•  Cost  of  care— total  and  its  components 

•  Percentages  of  target  populations  immunized 

•  Inhaler  teaching  documentation 

•  Waiting  time— for  appointments  and  at  appointments 

•  Blood  pressure  control 

•  Hyperlipidemia— percent age  with  cholesterol  check 

•  Rates  of  Pap  smear,  mammogram,  sigmoidoscopy  and  fecal  occult 
blood  testing 

•  Counseling— nutrition/ alcohol /tobacco 

•  Specialty  consults 

•  Mortality— risk-ad justed  death  rates  from  stroke,  myocardial 
infarction,  diabetes,  suicide,  flu  or  pneumonia 

•  Morbidity 

•  Preventable  adverse  drug  reaction  rate 

•  Admissions  for  asthma/diabetes  (preventable  by  aggressive 
management ) 

•  Wound  infection  rates 

•  Telephone  access/on-hold  t ime/interrupt ions  to  providers  or 
team  members 

•  Appointment  cancellations— pat ient-generated  vs.  clinic¬ 
generated  vs.  no-shows 
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Locally  Available  Data 

The  DVA  employs  numerous  data  collection  tools  on  a 
national  level  to  monitor  and  improve  performance  and  outcomes. 
The  validity  and  reliability  of  these  tools  is  monitored  at  the 
national  level.  National  customer  satisfaction  surveys,  a  DVA- 
specific  modification  of  the  SF-36  Health  Survey,  and  the 
Network  Directors'  Performance  Measures  are  among  national 
quality  initiatives.  Workload,  staffing,  and  cost  data  are  also 
collected  and  reported  throughout  the  system. 

The  customer  satisfaction  survey  is  administered  annually. 
The  validity  and  reliability  of  this  survey  instrument  are 
ensured  on  a  national  basis.  It  is  a  standardized  multiple- 
choice  questionnaire  designed  to  permit  analysis  of  trends  over 
time  and  comparison  with  DVA  and  external  benchmarks.  The 
external  benchmarking  tool  used  is  a  database  compiled  by  the 
non-profit  Picker  Institute  for  Patient-Centered  Care,  which 
surveys  academic  medical  facilities,  using  the  same  survey 
approach  and  questionnaire  used  by  the  DVA  (Office  of 
Performance  and  Quality,  1999).  Risk-adjustment  for  age,  sex, 
and  health  status  ensures  validity  of  scores  for  comparison.  The 
multiple-choice  questions  included  in  the  customer  satisfaction 
survey  represent  the  DVA' s  Customer  Service  Standards.  Questions 
are  grouped  into  domains  of  background,  demographics,  overall 
satisfaction,  access,  courtesy,  patient  educat ion/inf ormat ion, 
preferences,  emotional  support,  continuity  of  care,  overall 
coordination  of  care,  visit  coordination  of  care,  specialist 
provider  access,  and  pharmacy  access. 
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The  SF-36  is  a  36-item  general  health  status  assessment 
questionnaire  developed  as  part  of  the  Rand  Corporation's 
Medical  Outcomes  Study  (UCSD  HOAP,  1999) .  Its  reliability  and 
validity  have  been  substantiated.  The  Veterans  SF-36  is  an 
adaptation  of  the  SF-36  designed  specifically  for  use  with 
veterans  (Kazis  et  al,  1999) .  Data  collected  with  the  Veterans 
SF-36  are  reported  as  the  Preventative  Disease  Index,  Chronic 
Disease  Index,  and  Palliative  Index  components  of  the  Network 
Directors'  Performance  Goals. 

The  National  Performance  Management  Workgroup  develops  the 
Network  Director's  Performance  Measures,  with  analyses  by  the 
Office  of  Performance  and  Quality.  These  performance  measures 
have  been  designed  to  describe  and  measure  the  Network 
Directors'  mastery  of  executive  core  competencies,  utilization 
of  a  comprehensive  framework  for  assurance  of  quality  health 
care,  and  ability  to  deliver  healthcare  value,  as  well  as 
performance  in  areas  of  organizational  special  emphasis. 

An  Executive  Statistical  Report  is  prepared  at  the  Cheyenne 
VAMC  on  a  monthly  basis,  detailing  workload  and  revenue.  It 
includes  a  host  of  nationally  and  locally  generated  data 
including,  but  not  limited  to:  numbers  of  inpatients  treated; 
discharges;  deaths;  outpatient  visits;  and  total  unique,  i.e. 
individual,  patients  treated.  The  source  of  data  for  this  report 
is  VISTA  (Veterans  Health  Administration  Information  Systems 
Technology  Architecture) . 

Fiscal  service  tracks  and  trends  total  station  operating 
costs  and  the  components  that  contribute  to  that  total.  Major 
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contributors  to  total  cost  include  salaries  of  employees, 
pharmacy,  prosthetics,  utilities,  and  laboratory. 

Purpose  (Variables/Working  Hypothesis) 

The  purpose  of  this  project  is  twofold:  to  assess  whether 
sufficient  reliable  and  valid  data  exist  upon  which  to  base  a 
comparison  of  outcomes  before  and  after  the  implementation  of 
primary  care  teams;  and  if  so,  to  make  such  a  comparison. 

First  Null  Hypothesis:  Reliable  and  valid  data  are  readily 
available  for  comparison  of  outcomes  pre-  and  post-primary  care 
team  implementation. 

First  Alternate  Hypothesis:  Reliable  and  valid  data  are  not 
readily  available  for  comparison  of  outcomes  pre-  and  post¬ 
primary  care  team  implementation. 

Second  Null  Hypothesis:  Structure  and  process  changes 
resulting  from  primary  care  team  implementation  have  not 
significantly  impacted  outcomes. 

Second  Alternate  Hypothesis:  Structure  and  process  changes 
resulting  from  primary  care  team  implementation  have 
significantly  impacted  outcomes. 

Methods  and  Procedures 

Three  methods  have  been  employed  in  outcomes  research, 
including  prospective,  observational  studies  in  natural  clinical 
settings;  para-analyses  of  information  from  databases;  and  meta¬ 
analysis,  or  literature  review  (Generali,  1999). 
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A  para-analysis  following  the  JCAHO  Clinical  Value  Compass 
approach  was  chosen  to  assess  whether  the  implementation  of 
primary  care  teams  added  value  to  the  healthcare  provided  by  the 
Cheyenne  VAMC. 

A  measurement  tool  should  be  evaluated  using  three  major 
criteria:  validity,  reliability,  and  practicality.  Validity 
refers  to  the  extent  to  which  a  test  measures  what  the 
researcher  sets  out  to  measure.  It  may  be  determined  by  a  review 
panel.  Reliability  refers  to  the  accuracy  of  the  measuring 
procedure.  Practicality  refers  to  convenience,  economy,  and 
interpretability  (Cooper  &  Schindler,  1998). 

Potential  outcomes  measurements  were  identified  via  review 
of  the  literature.  A  panel  of  persons  involved  in  the  local 
collection  and  interpretation  of  data  reviewed  the  list  of 
identified  measurements.  As  the  list  was  extensive,  the  panel 
was  asked  to  assist  in  determining  which  measures  were  readily 
available  and  interpretable.  Panel  review  has  insured  that  only 
valid  and  practical  measures  were  selected. 

Further  refinements  reduced  the  selected  measures  to  a 
manageable  number,  in  alignment  with  the  Clinical  Value  Compass 
and  representing  each  of  its  four  cardinal  outcomes  categories. 
Table  one  summarizes  the  measures  chosen  for  each  category. 
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Table  1 

JCAHO  Clinical  Values  Compass -Out comes  Selection 


Cardinal  Points 


Functional 

Clinical 

Status 

Costs 

Satisfaction 

Outcomes 

0 

u 

t 

Ambulatory- 

Procedures 

Total  Station 
Cost/Patient 

Continuity 
of  Care 

OP  Visits/ 
Patient  Treated 

c 

o 

m 

Preventative 

Disease  Index 

Pharmacy  Cost/ 
Patient 

Coordination 

of  Care 

Pt  Days  of  Care/ 
Patient  Treated 

e 

s 

Chronic 

Disease  Index 

Courtesy 

Inpatients 

Treated/ 

Patient 

M 

e 

Education 

Patient  Death 

Rate 

a  Emotional 

s  Support 


u 

r  Preferences 

e 

s  Timeliness/ 

Access 


Customer 
Service 
Standards — 

Average 

Note.  Cardinal  points  correspond  to  four  categories 
of  outcomes. 
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The  time  period  covering  FY-96  through  FY-00  was  selected 
for  data  analysis.  Since  primary  care  team  implementation 
occurred  during  the  first  quarter  of  FY-99,  data  from  FY-99 
through  FY-00  were  determined  to  best  represent  the  post¬ 
implementation  timeframe.  FY-96  through  FY-98  was  selected  as 
the  pre-implementation  control  period.  Paired  t-tests  were  used 
to  measure  significance  of  differences,  and  the  95%  confidence 
interval  was  chosen  as  appropriate  (p  <  .05) . 

Data  were  extracted  from  customer  satisfaction  survey 
results.  Network  Directors'  Performance  Measures,  fiscal 
reports,  and  statistical  summaries  for  the  pre-  and  post-primary 
care  team  implementation  timeframes.  Data  analysis  compared 
outcomes  measures  in  an  attempt  to  determine  whether  the  change 
to  primary  care  teams  led  to  changed  outcomes . 

Results 

Total  station  cost  per  patient  was  lower  during  the  post¬ 
implementation  timeframe  than  during  the  pre-implementation 
timeframe.  The  difference  was  statistically  significant  when 
adjusted  for  inflation  based  on  the  consumer  price  index  (CPI) . 
Table  two  provides  this  information,  which  is  graphically 
illustrated  in  figure  two. 
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Table  2 

Total  Station  Cost  per  Patient 


FY-96 

FY-97 

FY-98 

FY-99 

FYOO-Proj 

P 

Total  Cost  /  Patient 

$  3,487 

$  3,725 

$  3,204 

$  3,175 

$  3,161 

0.0903 

Adjusted  Cost  / 
Patient 

$  3,487 

$  3, 617 

$  3,020 

$  2,906 

$  2,808 

0.0478 

Total  and  Adjusted  Station  Cost 
Per  Patient 


$4,  0 


$3,5 


$3,  0 


$2,5 


Total 
Cost  / 
Patient 


Adjusted 
Cost  / 
Patient 


Figure  2 .  Total  Station  Cost  per  Patient 
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Pharmacy  cost  per  patient  increased  post-implementation  of 
primary  care,  as  shown  in  table  three  and  figure  three.  However, 
the  increased  cost  was  not  statistically  significant  when 
adjusted  for  inflation  based  on  the  CPI. 


Table  3 

Pharmacy  Cost  per  Patient 


FY-96 

FY-97 

FY-98 

FY-99 

FYO 0-Pro j 

P 

Pharmacy  Cost/ 

Patient 

$ 

233 

$ 

362 

$ 

387 

$ 

475 

$ 

486 

0.0414 

Adjusted  Cost/ 

Patient 

$ 

233 

$ 

352 

$ 

365 

$ 

434 

$ 

432 

0.0541 

Figure  3 .  Pharmacy  Cost  per  Patient 
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Customer  satisfaction  results  were  mixed,  as  shown  in 
table  four  and  figure  four.  It  should  be  noted,  for 
interpretation  of  these  figures  and  tables,  that  a  lower  number 
indicates  a  more  desirable  outcome.  Values  shown  are  percentages 
of  customers  reporting  problems  in  each  domain  of  satisfaction. 
Emotional  support,  continuity,  education,  and  preference  were 
essentially  unchanged.  Courtesy,  timeliness/access,  and  overall 
customer  satisfaction  survey  average  showed  trends  indicating 
improvement,  but  the  changes  were  not  statistically  significant. 
Improved  coordination  of  care  was  the  only  change  of  a 
statistically  significant  magnitude. 

Table  4 

Customer  Satisfaction  Survey  Results 


FY-96 

FY-97 

FY-98 

FY-99 

P 

Continuity 

30.00% 

19.00% 

32.2  0% 

2  8.2  4% 

0.4  0  0  4 

Coordination 

of  Care 

33.00% 

2  9.0  0% 

19.00% 

12.00% 

0.0  34  6 

Courtesy 

15.00% 

7.00% 

8.90% 

7.27% 

0.1680 

Education 

27.00% 

2  3.0  0% 

2  8.7  0% 

24.7  4% 

0.2350 

Emotional 

Support 

18.00% 

13.00% 

18.60% 

12.51% 

0.0  7  5  8 

Preferences 

18.00% 

17.00% 

2  3.8  0% 

18.55% 

0.334  7 

Timelines  s / 

Acce  s  s 

19.00% 

12.0  0% 

10.80% 

8.94% 

0.0  95  0 

Customer 

Service 

Standards — 
Average 

22.86% 

17.14% 

2  0.2  9% 

16.04% 

0.0  6  67 

Note.  Lower  number  is  desirable 
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Customer  Satisfaction  Survey  Results 

(Lower  Value  is  Desirable) 

35.00% 

30.00% 

25.00% 

20.00% 

15.00% 

10.00% 

5 .00% 

FY-96  FY-97  FY-98  FY-99 


Figure  4  a.  Customer  Satisfaction  Survey  Results 
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Figure  4  b.  Customer  Satisfaction  Survey  Results 

Functional  status  measures  improved  post-implementation  of 
primary  care,  as  displayed  in  table  five  and  figure  five. 
However,  none  of  the  changes  were  statistically  significant. 
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Table  5 

Functional  Status  Indices 


FY-96 

FY-97 

FY-98 

FY-99 

Qtrl- 

FYOO 

P 

Ambulatory  Procedures 

65% 

89% 

88% 

96% 

0.0948 

Preventative  Disease  Index 

35% 

73% 

89% 

93% 

97% 

0 . 1035 

Chronic  Disease  Index 

38% 

66% 

91% 

92% 

99% 

0.0899 

Figure  5 .  Functional  Status  Indices 


Clinical  outcomes  also  showed  a  trend  indicating 
improvement  coincident  with  primary  care  implementation.  Tables 
six  through  nine  and  figures  six  through  nine  illustrate 
clinical  outcomes  measures.  Although  there  was  a  statistically 


Structure--Process--Outcome 


31 


significant  increase  in  outpatient  visits  per  patient,  the 
number  and  length  of  hospitalizations  and  the  number  of  deaths 
per  1000  patients  treated  all  decreased  to  a  somewhat  less  than 
statistically  significant  magnitude. 


Table  6 

Outpatient  Visits  per  Patient  Treated 


FY-96 

FY-97 

FY-98 

FY-99 

FY-00 

Proj 

P 

CP  Visits/ 
Patient  Treated 

7.84 

8.19 

8.16 

8.91 

9.00 

0.0045 

Total  Patients 

Treated 

6,589 

6,653 

8,002 

9,116 

10,000 

Total  OP  Visits 

51,651 

54,491 

65,290 

81,262 

90,000 

Figure  6.  Outpatient  Visits  per  Patient  Treated 
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Table  7 

Patient  Days  of  Care  per  Patient  Treated 


FY-96 

FY-97 

FY-98 

FY-99 

FY-00 

Proj 

P 

PDOC/Patient 

Treated 

1 . 65 

0.90 

0.67 

0.59 

0.49 

0 . 1041 

Total  Patients 

Treated 

6,  589 

6,  653 

8,  002 

9,  116 

10,000 

Total  PDOC 

10, 892 

6,  009 

5,  371 

5,  342 

4,884 

Figure  7 .  Patient  Days  of  Care  per  Patient  Treated 
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Table  8 

Inpatients  treated  per  Patient 


FY-96 

FY-97 

FY-98 

FY-99 

FY-00 

Proj 

P 

Inpatients 
Treated  /Patient 

0.23 

0.13 

0 . 11 

0.10 

0  .10 

0.1304 

Total  Patients 

Treated 

6,589 

6,  653 

8,002 

9,  116 

10,000 

Inpatients 

Treated 

1,513 

885 

906 

923 

1,  044 

Figure  8 .  Inpatients  Treated  per  Patient 
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Table  9 

Patient  Death  Rate 


FY-96 

FY-97 

FY-98 

FY-99 

FY-00 
Pro  j 

P 

Deaths/  1000 
Patients  Treated 

5.3119 

2 .4049 

3.1242 

2.3036 

2 . 1000 

0 . 1233 

Patients  Treated 

6,  589 

6,  653 

8,  002 

9,  116 

10,000 

Total  Deaths 

35 

16 

25 

21 

21 

Figure  9.  Patient  Death  Rate 
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Discussion 

For  the  JCAHO  Clinical  Value  Compass  cardinal  point 
corresponding  to  cost,  total  station  cost  per  patient  treated 
was  selected  as  the  most  meaningful  indicator  of  direct 
healthcare  costs.  Indirect  costs  of  care  such  as  patient  time 
lost  from  work  could  not  be  quantified.  Total  station  cost  per 
patient  treated,  adjusted  for  inflation,  decreased  significantly 
after  primary  care  team  implementation.  Costs  were  expressed  in 
terms  of  1996  dollars  in  order  to  make  a  valid  comparison  over 
time.  The  CPI  was  used  as  an  adjustment  factor.  Health  care  cost 
inflation  has  historically  exceeded  CPI  inflation,  but  figures 
on  health  care  inflation  were  not  consistent  among  sources,  and 
disparity  existed  between  values  reported  for  the  private  and 
public  sector.  Had  reliable  healthcare  inflation  rate 
information  been  available  for  use,  the  decrease  in  costs  would 
probably  appear  more  highly  significant  than  the  values 
presented . 

To  demonstrate  that  cost  changes  in  a  given  area  are  not 
always  directly  related  to  change  in  total  station  cost, 
pharmacy  cost  was  included  in  this  study.  The  pharmacy  cost  per 
patient  treated  increased,  but  the  increase  was  not 
statistically  significant  when  adjusted  for  inflation.  The 
increase  in  pharmacy  cost  coupled  with  decreased  total  station 
cost  per  patient  treated  lends  credence  to  the  concept  that 
intensive  appropriate  pharmaceutical  intervention  can  save 
healthcare  funds  by  keeping  patients  out  of  the  hospital. 
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The  cardinal  compass  point  of  satisfaction  was  readily 
reviewable  based  on  annual  customer  satisfaction  survey  results. 
Seven  domains  of  satisfaction  have  been  consistently  surveyed 
throughout  the  FY96-00  timeframe.  These  include  continuity  of 
care,  coordination  of  care,  courtesy,  education,  emotional 
support,  preferences,  and  timeliness/access .  An  average  of  these 
seven  measures  was  calculated  to  show  trends  from  a  more  global 
perspective.  Coordination  of  care  was  significantly  improved 
after  primary  care  team  implementation,  and  all  other  survey 
results  were  either  unchanged  or  improved,  but  not  to  a 
statistically  significant  degree. 

One  would  expect  continuity  of  care  to  be  higher  with 
primary  care  than  without.  However,  it  remained  essentially 
unchanged  when  pre-  and  post-implementation  timeframes  were 
compared.  Subject  matter  experts  at  the  medical  center  indicated 
that  continuity  of  care  decreased  in  FY-98  due  to  "growing 
pains"  associated  with  the  opening  of  community-based  outpatient 
clinics  in  northern  Colorado.  Shifting  of  permanent  staff  to  the 
additional  clinics  was  a  slow  process.  Now  that  permanent  staff 
is  in  place,  continuity  of  care  scores  are  dropping,  indicating 
increased  customer  satisfaction  with  this  aspect  of  care. 

Functional  status  measures  consistently  available  for  the 
study  timeframe  included  the  SF-36  based  Preventive  Disease 
Index  and  Chronic  Disease  Index,  and  the  percentage  of 
procedures  done  in  the  ambulatory  care  setting.  Trends  toward 
improvement  were  evident  for  each  measure,  but  did  not  reach 
statistically  significant  levels. 
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Change  in  the  traditional  clinical  outcomes  of  morbidity 
and  mortality  are  difficult  to  measure  in  the  short-term. 
Therefore,  surrogate  measures  of  morbidity  were  used  in  this 
study.  Outpatient  visits  per  patient  treated  increased 
significantly  post-implementation,  which  may  be  indicative  of 
either  an  increase  in  morbidity  or  newly  increased  emphasis  on 
preventative  healthcare.  Decreased  utilization  of  inpatient 
hospital  care  may  indicate  decreased  morbidity.  This  is  also  a 
desirable  trend  from  a  resource  management  point  of  view,  as  it 
represents  a  shift  from  more  resource-intensive/expensive 
inpatient  care  to  less  expensive,  less  intensive  care  delivered 
in  the  outpatient  setting.  Due  to  this  shift  in  care  delivery, 
the  Cheyenne  VAMC  has  been  able  to  treat  an  appreciably  greater 
number  of  patients  without  hiring  additional  staff,  as 
illustrated  by  Appendix  B. 

Deaths  per  1000  patients  treated  also  decreased  pre-  to 
post-primary  care  team  implementation.  However,  it  should  be 
noted  that  this  figure  includes  only  deaths  that  occurred  in  the 
VAMC,  so  is  not  a  true  measure  of  total  population  mortality. 
This  indicator  should  be  viewed  with  this  understanding. 

Conclusion  and  Recommendations 

Reliable,  valid  outcomes  measures  were  identified  among 
currently  collected  information.  Outcomes  in  all  four  of  the 
cardinal  points  of  the  Clinical  Value  Compass  were  improved  or 
unchanged.  An  association  was  demonstrated  between  outcomes 
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changes  and  primary  care  team  implementation.  However,  the 
validity  of  this  association  may  be  questioned.  According  to  the 
concept  of  maturation,  changes  within  a  population  studied  may 
be  attributable  to  passage  of  time  rather  than  to  a  specific 
change.  This  can  be  a  threat  to  experimental  validity  (Cooper  & 
Schindler,  1998) .  Since  the  healthcare  environment  is  far  from 
static,  other  changes  such  as  improved  technology  may  be 
considered  as  confounding  variables  in  this  study,  and  multiple 
variables  may  have  contributed  to  changes  in  selected  outcomes 
measures . 

The  first  null  hypothesis,  that  reliable  and  valid  data 
are  readily  available  for  comparison  of  outcomes  pre-  and  post¬ 
primary  care  team  implementation,  was  not  rejected.  The  first 
alternate  hypothesis,  that  reliable  and  valid  data  are  not 
readily  available  for  comparison  of  outcomes  pre-  and  post¬ 
primary  care  team  implementation,  was  rejected.  Reliable  and 
valid  data  were  identified  upon  which  to  base  outcomes 
comparisons.  A  decision  to  reject  or  fail  to  reject  was  not 
reached  for  either  the  second  null  hypothesis  (structure  and 
process  changes  resulting  from  primary  care  team  implementation 
have  not  significantly  impacted  outcomes)  or  the  second 
alternate  hypothesis  (structure  and  process  changes  resulting 
from  primary  care  team  implementation  have  significantly 
impacted  outcomes).  A  positive  association  has  been  identified 
between  outcomes  and  primary  care  team  implementation,  but 
attributing  the  changed  outcomes  to  primary  care  team 
implementation  cannot  be  done  with  any  degree  of  certainty. 
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Review  of  the  valid,  reliable  measures  identified  in  this 
project  could  be  continued/updated  to  measure  success  or  lack 
thereof  for  other  new  programs,  in  addition  to  ongoing  review  of 
primary  care.  Results  could  be  applied  to  VAMCs  of  similar  size 
and  scope,  and  the  study  could  be  replicated  at  other  size  and 
scope  institutions  for  comparison/applicability.  If  the  study 
were  replicated  at  an  institution  of  similar  size  and  scope,  but 
lacking  primary  care  teams,  perhaps  a  comparison  would  lead  to 
more  definitive  conclusions. 
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Appendix  A 

Definitions  of  Acronyms  and  Terms 


CBOC 

CD  I 

CHF 

CPI 

CSS 

DRG 

DSS 

DVA 

FTEE 

FY 

GEM 

HALE 

HRQOL 

JCAHO 

OP 

Outcomes  measures 

PCP 

PDOC 

PD  I 

QI 

SF-36 


Community-Based  Outpatient  Clinic 
Chronic  Disease  Index 
Congestive  Heart  Failure 
Consumer  Price  Index 
Customer  Satisfaction  Survey 
Diagnosis  Related  Group 
Decision  Support  Service/System 
Department  of  Veterans  Affairs 
Full  Time  Employee  Equivalent 

Fiscal  Year  (DVA' s  is  October  1-  September  30) 

Geriatric  Evaluation  and  Management 

Health-adjusted  Life  Expectancy 

Health  Related  Quality  of  Life 

Joint  Commission  on  Accreditation  of 

Healthcare  Organizations 

Outpatient 

Indicators  of  population  health 
Primary  Care  Provider 
Patient  Days  of  Care 
Preventative  Disease  Index 
Quality  Improvement 

Short  Form  36,  Rand  Corporation's  36-item 
general  health  status  assessment  questionnaire 


Unique  Patient 

VAMC 

VISTA 
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Individual  patient  being  treated  at  the 
medical  center  during  a  given  fiscal  year 
Veterans  Affairs  Medical  Center 
Veterans  Health  Administration  Information 
Systems  Technology 
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Appendix  B 


Workload  and  Staffing  by  Fiscal  Year 


Employees 
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